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Abstract 

We  provide  a  necessary  and  sufficient  condition  for  the  almost  sure  conver¬ 
gence  and  the  strong  consistency  of  the  sample  autocovariance  of  a  discrete 
spectrum  weakly  stationary  process.  This  also  clarifies  the  estimation  of  the 
autocovariance  function  of  a  mixed  spectrum  weakly  stationary  processes. 


Abbreviated  Title:  “Autocovariance” 


Key  words  and  phrases:  Mixed  spectrum,  SLLN,  stationary,  sample  co¬ 
variance,  almost  sure,  amplitude,  phase,  zero-crossing  rate. 


1  Introduction 

The  estimation  problem  of  the  autocovariance  function  of  a  weakly  station¬ 
ary  process  is  of  fundamental  importance  in  time  series  analysis,  and  has 
been  treated  in  the  past  by  many  authors.  Most  of  the  literature  deals  with 
the  continuous  spectrum  case  where  the  results  depend  on  a  vanishing  auto¬ 
correlation,  and  the  existence  of  higher  order  moments.  The  more  complex 
mixed  spectrum  case,  however,  has  received  less  attention  and  is  less  clear. 
The  source  of  confusion  is  the  fact  that  in  the  presence  of  a  discrete  spec¬ 
tral  component  the  usual  sample  estimator  is  not  necessarily  consistent.  For 
example,  for  a  Gaussian  process,  the  sample  autocovariance-the  estimator 
most  often  used  in  practice-is  inconsistent  in  mean  square  sense  unless  the 
discrete  spectrum  is  absent  (Hannan  (1970),  p.  210,  Koopmans  (1974,  p. 
60)).  Moreover,  in  general  consistency  results  concerning  the  sample  autoco¬ 
variance  are  not  entirely  satisfactory.  The  purpose  of  this  note  is  to  clarify 
the  general  estimation  problem  in  the  mixed  spectrum  case  vis-a-vis  the  al¬ 
most  sure  convergence  of  the  sample  autocovariance  under  the  assumption 
of  second  order  stationarity  only.  In  a  nutshell,  this  is  achieved  by  observing 
that  the  product  of  the  process  by  its  shift  results  in  a  process  which  is  the 
Fourier  transform  of  an  L 1  random  measure. 

The  general  mixed  spectrum  case  encountered  in  practice  is  that  of  “signal 
plus  noise”  where  the  signal  is  a  stationary  process  consisting  of  a  sum  of 
random  sinusoids,  and  the  noise  is  a  stationary  process,  with  certain  ergodic 
properties,  independent  of  the  signal.  With  this  in  mind,  it  is  more  crucial 
to  consider  the  discrete  spectral  component. 

To  motivate  our  discussion,  consider  the  strictly  stationary  real-valued 
process 

X,  =  !>•'"%,  = 

i 

where  the  fj  are  orthogonal.  It  is  shown  in  Hannan  (1970,  p.  201)  that  a 
necessary  condition  for  ergodicity  is  that  the  |£j|  are  almost  sure  constants. 
Clearly,  under  ergodicity  the  sample  autocovariance  is  strongly  consistent. 
Without  addressing  ergodicity  directly,  we  shall  show  that,  under  a  certain 
assumption  on  the  frequencies,  constant  amplitudes  are  necessary  and  suffi¬ 
cient  for  the  strong  consistency  at  all  lags  of  the  sample  autocovariance  for 
any  complex-valued  weakly  stationary  discrete  spectrum  process.  The  ar¬ 
guments,  which  makes  use  of  a  new  work  on  the  SLLN  by  Houdre  (1992), 
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involves  second  order  properties  only.  These  arguments  are  given  here  only 
in  the  discrete  parameter  case,  but  the  continuous  and  the  multidimensional 
parameter  cases  follows  similarly  (the  corresponding  strong  laws  are  proved 
in  the  above  reference. 


2  A  SLLN 


We  shall  appeal  to  the  following  general  result  which  states  that  the  strong 
law  of  large  numbers  (SLLN)  holds  for  processes  which  are  Fourier  transforms 
of  random  measures  provided  the  latter  converges  almost  surely  to  zero  at 
punctured  dyadic  intervals  around  the  origin. 

Theorem  2.1  Suppose  {W},-  t  —  0,±1,---,  admits  the  Fourier  representa¬ 
tion  v 

Xt  =  [  e^tidu,) 

J  —TV 

where  £  :  £>(— 7r,7r]  — »  La(P),  1  <  a  <  2,  is  cr-additive.  Then 


&  £  £*.  =  «{«}) 


t= l 


in  La(P).  Furthermore , 


lim 

N—yoo 


1 

N 


N 


£**  =  £({»}) 


a.s. 


if  and  only  if 

lim  £((— 2_”,2_ri)  \  {0})  =  0  a.s. 

n—yoo 


This  is  a  generalization  of  a  result  by  Gaposhkin  (1977)  who  requires  weak 
stationarity.  It  seems  that  the  precursor  to  all  this  is  a  result  of  Rajchman 
(1932)  which  states  that  the  strong  law  of  large  numbers  holds  for  a  sequence 
of  uncorrelated  random  variables  whose  second  moments  have  a  common 
bound  (see  Chung  (1974,  p.  103)).  The  more  general  dependent  case  is 
discussed  in  Chapter  10  of  Doob  (1953). 

We  note  that  the  dc  component  £({0})  may  be  random  or  nonrandom, 
and  that  we  do  not  require  the  mean  of  {Xt }  to  be  zero. 
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The  proof  of  Theorem  2.1  depends  on  a  careful  analysis  in  a  neighbor¬ 
hood  of  the  origin  of  the  kernels  /i;v(d)  =  4  J2tLi  exp(it6).  Since  M*) is 
periodic  with  period  27T,  Theorem  2.1  has  to  be  modified  when  the  limits 
of  integration  extend  beyond  (— 7r,7r]  and  cover  multiples  of  ±27i\  In  this 
case,  in  addition  to  the  origin,  we  must  examine  neighborhoods  of  multiples 
of  ±27T.  In  particular,  when  the  limits  extend  from  — 27r  to  27t,  the  modified 
claim  is 

Theorem  2.2  Suppose  { Xt },  t  =  0,  ±1,---,  admits  the  Fourier  representa¬ 
tion 

/2tt 

eitw£(du>) 

-2  TT 

where  £  :  B(— 27r,27r]  — »■  La{P),  1  <  a  <  2,  is  a-additive.  Then 

1™  ={({<>» +«{2*}) 

N-+oo  J\ 

in  La(P).  Furthermore, 

1  N 

^r^Xt  =  ^({°})  +  ^({27rl)  a-s ■ 

N—koo  2\  “ 

if  and  only  if 

{{((-2'”.  2-”)  \  {0})  +  f(-2x,  -2*  +  2-)  +  «2x  ,  2“",  2x)}  =  0 

almost  surely. 

This  consideration  becomes  relevant  in  the  proof  of  strong  consistency  of  the 
sample  covariance.  The  proof  of  Theorem  2.2  follows  as  in  Houdre  (1992) 
where  the  case  of  a  spectrum  supported  on  (— 7r,7r]  is  tackled.  It  is  also  a 
special  case  of  a  general  a.s.  principle  developed  in  Houdre  (1993). 

There  are  various  applications  in  spectral  analysis  where  the  almost  sure 
result  concerning  the  sample  autocovariance  is  of  use.  For  example,  in  the 
mixed  spectrum  case,  when  precise  autocovariance  estimates  are  available, 
it  is  possible  to  detect  the  discrete  frequencies  from  the  sample  covariance 
evaluated  at  sufficiently  high  lags  (Priestley  (1981),  p.  626).  More  recently,  it 
has  been  shown  in  Li  and  Kedem  (1993)  that  a  certain  parametrization  of  the 
sample  autocorrelation  forms  a  contraction  mapping  whose  set  of  fixed  points 
contains  approximations  to  the  cosines  of  the  true  frequencies.  The  method, 
which  leads  to  very  precise  frequency  estimates,  requires  however,  among 
other  things,  the  almost  sure  convergence  of  the  sample  autocovariance. 
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3  The  Discrete  Spectrum  Case 


3.1  Finitely  Many  Frequencies 

Let  {Xt},  t  =  0,  ±1,  ±2,  •  •  •,  be  a  weakly  stationary  discrete  spectrum  process 
with  mean  E[Xt]  =  m ,  autocovariance  R(k )  =  E[(Xt+k  —  m)(Xt  —  m)\, 
k  =  0,  ±1,  ±2,  •  •  •,  and  spectral  distribution  F(duj)  supported  at  p+1  distinct 
atoms  in  •  •  •  ,cop}  £  (— 7r,  7r] ,  such  that  u>o  =  0.  Then, 

Xt  -  m  =  r  eitw({du) 

J  —  7T 


where, 

«(<M  = 

i=o  j=o 

and  the 

})  =  & 

are  orthogonal  with  mean  zero.  Clearly  £  :  B{— 7r,7r]  — *  L2(P)  is  a-additive. 


3.1.1  Estimation  of  the  Mean 


As  an  illustration  of  the  use  of  Theorem  2.1,  consider  the  estimation  problem 
of  the  mean  E[Xt]  =  m.  Write 

Xt  =  f  e^(i(du)  +  m60(du)) 

J  —71 

Since  the  loj  are  distinct,  for  sufficiently  large  n, 


a(-2~n,2~n)  \  {0})  +  mS o((-2-n,2-n)  \  {0})  =  0  a.s. 


Therefore, 

(i) 


SSLJi  S*  =  «{°})  + 


m  a.s. 


and  we  obtain  the  well  know  result  that 


1 


N 


lim  —  V  Xi  =  m 

N—>oo  N  “[ 
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if  and  only  if  £({0})  =  0  almost  surely.  Observe  that  the  dc  component 
£({0})  +  m  in  the  second  order  stationary  case  is  a  sum  of  a  random  part 
mO})  plus  a  nonrandom  constant  m.  Clearly,  “£({0))  +  m”  here  plays  the 
role  of  “£({0})”  in  Theorem  2.1. 

3.1.2  Estimation  of  the  Autocovariance 

We  shall  treat  the  general  case  which  takes  into  account  the  presence  of 
a  possible  dc  component.  That  is,  we  do  not  necessarily  require  m  —  Cl 
and/or  £({0})  =  0  a.s.  .  In  practice  however,  the  dc  component  is  often 
removed  by  subtracting  the  sample  average  in  accordance  with  the  SLLN 
(  1)  (see  Koopmans  (1974),  p.58).  As  an  estimator  for  R(k)  we  choose  the 
(simplified)  sample  autocovariance  (1/N)  J2tLi(Xt+k  —  m)(X h  —  m). 

If  we  define 

=  { E  |{({^})|2e*"<  ]  6o(Jw)  + 

U=°  ) 

Then 

/27T 
-27 r 

and  we  can  see  that  for  each  fixed  k,  the  lag  process  (Xt+k  —  rn)(Xt  —  m) 
admits  a  Fourier  representation  with  respect  to  ^  :  Z?(— 27r,27r]  — >  Ll{P), 
and  a  dc  component 


e<t,({o})  =  Eif(K})iv'‘“' 

3=0 

Notice  the  emergence  of  a  dc  component  in  the  lag  process  (Xt+k—m,)(Xt  —  m) 
regardless  of  whether  it  is  absent  or  present  in  the  original  process  {Ah}. 

Since  the  Uj  are  distinct,  there  is  no  •)  mass  in  sufficiently  small 

punctured  neighborhoods  of  0  and  of  ±27t.  In  addition,  £^({27r})  =  0. 
Therefore,  by  Theorem  2.2 

i  w  _ 

-  E(V,+l  -  m)(X,  -  m)  H  Vi 

t  =  1 
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We  need  conditions  to  ensure  that  £^({0})  is  a  nonrandom  constant  for  all 
k,  in  which  case 

{(t»({0})  =  £[{<‘>({0})]  a.s„  W 
Clearly,  £[{«({0})]  =  R(k). 

So,  assume  ^fc^({0})  =  i?[£^({0})],  k  —  0,  ±1,  •  •  •.  Then  for  all  &, 


E(I6|2-^IV^=0 


i=o 


and  this  implies  that 


or, 

i  =  o,  i,---, p 

where  the  <pj  are  random  such  that  the  are  orthogonal  with  mean  0. 

On  the  other  hand,  when  —  \J E\^} |2e*^ ,  then  |£j  p  =  E\£j\2  and 
£^({0})  =  f£[£^({0})]  =  ft(k)  is  a.s.  nonrandom  for  all  k.  We  thus  have. 

Theorem  3.1  Let 

(2)  Xt  =  m  +  j2eiu,%  i  =  0,±1,±2,--- 

i= o 

where  E[(f]  —  0,  and  for  j  /  l  E[^j(i]  =  0 ,  be  a  complex  valued  weakly  sta¬ 
tionary  process  with  mean  m,  autocovariance  R(-),  and  a  spectrum  supported 
at  p  +  1  distinct  atoms  {o>0  =  0,u>i,  •  •  •  ,  u;p}  G  (— 7r,  7t]  .  Then  as  N  — »  oo, 

(3)  T7E(V<+l'-™)(A',-m)L4  '{W({0)),  k  =  0,  ±1,  ±2,  ■  ■  ■ 

iV  t= 1 

Furthermore, 

i  iv  _  j 

W  «  £(*<+*  -  -RW,  t  =  o,±i,±2,--- 

iV  t=l 

if  and  only  if 

(5)  &  =  v^l&l2e*0>>  i  = 

where  the  <f>j  are  random  phases. 
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As  a  corollary  of  Theorem  3.1  we  can  see  that  if  {Ah}  in  (  2)  is  a  real¬ 
valued  Gaussian  process,  its  sample  autocovariance  is  not  consistent.  This 
is  so  because  of  the  requirement  of  constant  amplitudes  |£j|.  This  is  in  line 
with  the  well  known  fact,  mentioned  earlier,  that  the  presence  of  a  discrete 
spectrum  renders  the  sample  autocovariance  of  a  Gaussian  process  inconsis¬ 
tent  in  X2(P)  sense. 

Example  1.  Consider  the  model  of  random  phases 

p 

Xt  =  ^2  A i  COs(wjt  +  t  =  0,  ±1,  ±2  •  •  • 

j- 1 

where  E[Xt\  —  m  —  0,  0  <  uq  <  o>2  <  •  •  •  <  ujp  <  t,  Aj  are  positive 
constants,  and  the  < pj  are  independent  random  variables  uniformly  distributed 
in  ( — 7T ,  7r ] .  Define 

(f>-j  =  —  <f>j,  u>-j  =  Ljj ,  =  Aj,  Aq  —  0 

Then  we  have  the  representation 

p  1 

A't  =  ~A3el4>>e™A 

j=-p  “ 

so  that  (1/N)  J2tLi  Xt+kXt  R(k).  From  this,  the  sample  autocorrelation 
is  a.s.  consistent  as  well.  □ 


3.2  Infinitely  Many  Frequencies 

This  case  takes  after  the  case  of  finitely  many  frequencies  with  the  added 
assumption  that  for  j  /  /,  | iqj  —  <u/|  cannot  be  too  close  to  0  and  ±27r.  At 
this  point  it  might  be  worthwhile  to  mention  that  the  case  of  infinitely  many 
frequencies  is  also  important  in  practice:  Simple  nonlinear  operations  on  a, 
sum  of  pure  random  sinusoids,  e.g.,  clipping,  lead  to  a  discrete  spectrum 
process  with  infinitely  many  frequencies  (see  [7]). 
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Suppose  that  for  c Oj  6  ( —  7r,  7r] , 

OO 

(6)  A',  =  m  +  £ 

j=  o 

is  weakly  stationary  with  mean  m.  Then  by  stationarity  the  are  orthogonal 
and 

OO 

£|v.P  =  £I6I2<°° 

>  c 

It  follows  that 

(Xt+k  -  m)( Xt  -  m)  =  /  e^C^(doj) 

J -2  ir 

where 

OO  OO 

3=0 1=0 

is  cr-additive  from  B(— 2tt,27t)  to  L1(P)  since  Y,j  |£j|2  <  Thus  for  each 
fixed  k ,  the  lag  process  {(Xt+k  —  m)(Xt  —  m)},  t  —  0,  ±1,  ±2,  •  •  •,  admits  a 
Fourier  representation  with  respect  to  with  a  dc  component 

OO 

fw({o})  =  Efclv‘"’ 

3=0 

Note  that  since  — 7r  <  Uj  <  n,  we  always  have  —2 n  <  uj  —  u>i  <27 r  for 
distinct  coj,coi,  so  that  £^({27r})  =  0.  Therefore,  by  Theorem  2.2  and  the 
previous  discussion  on  equality  between  £^({0})  and  R(k ),  we  have. 

Theorem  3.2  Let  {Xt},  t  =  0,  ±1,-  •  be  a  complex  valued  weakly  station¬ 
ary  process  with  mean  m,  autocovariance  R(-),  and  a  spectrum  supported  at 
infinitely  many  distinct  atoms  u>j  £  ( — 7r ,  7r]  as  in  (  6).  Then  as  N  — >  oo, 

(7)  i  Y.(X,+t  -  m)( X,  -  m)  ii  £“>({0}) 

t= 1 

and  the  convergence  holds  almost  surely  if  and  only  if  as  n  — >  oo 

[£(fc)((— 2-n,2-n)\{0})  +  e(fc)(-27r,-27T  +  2-n) 

(8)  +  ^fc)(27r-2-n,27r)]“-4-0 
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Furthermore ,  when  (8)  holds, 

(9)  f  £(A'f+i  -  m)(X,  -  m)  "-4‘  fl(fc) 

iV  t=l 

for  all  k  (=  0,  ±1,  ±2,  •  •  ■),  if  and  only  if 

(10)  C  =  )=  0,1,2,--- 

where  the  <pj  are  random  phases. 

What  Theorem  3.2  says  in  effect  is  that  if  the  frequency  differences  cluster 
in  the  critical  intervals  (—2"”,  2-n)  \  {0}),  (—27 r,  —2ir  +  2~n),  (2n  —  2~n ,  27r), 
the  almost  sure  convergence  might  not  hold.  Evidently,  a.s.  convergence 
holds  when  inf  \u)j  —  u>i\  >  0  and  sup^,  \uj  —  u>/|  <  2i r.  Once  the  almost 
sure  convergence  occurs,  the  result  also  provides  a  necessary  and  sufficient 
condition  for  the  limit  to  be  a  constant. 


3.2.1  Connection  With  the  Zero-Crossing  Rate 

Theorem  3.2  clarifies  the  asymptotic  behavior  of  the  observed  zero-crossing 
rate. 

Let  {Ztj  be  a.  real-valued  stationary  Gaussian  process  with  mean  0,  and  a 
purely  discrete  spectrum  containing  a  finite  number  of  jumps  (atoms).  Also, 
let  Xt  =  1  when  Zt  >  0  and  Xt  =  0  otherwise.  We  observe  that  the  spectrum 
of  {W}  is  an  infinite  sum  of  convolutions  in  terms  of  the  spectrum  of  { Zt }, 
so  that  {W}  admits  the  representation  (  6)  with  E[Xt)  =  m  =  1/2. 

The  zero-crossing  rate  (ZCR)  7  (in  discrete  time)  observed  in  a  time  series 
Z\,  Zi,  •  •  •  ,  Z/v  is  defined  in  terms  of  the  indicators  dt  =  (Xt  —  W-i)  , 


7  = 


D 

N  -  1 


1 

iV  -  1 


AT 

£<f< 


t= 2 


Thus,  asymptotically  the  ZCR  is  the  same  as 

7  =  I  -  f  SW  -  1  -  m) 

Z  iV  <=2 
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where  E[Xt\  —  m  =  1/2,  and  we  can  study  the  asymptotic  ZCR  through 
the  first  order  sample  autocovariance  of  {Ah}.  So,  does  the  ZCR  converge 
almost  surely  to  a  constant  ?  This  is  addressed  in  Kedem  and  Slud  (1993). 
Let  <Td(-)  denote  the  spectral  measure  of  the  binary  process  {dt). 

Theorem  3.3  (Kedem  and  Slud  (1993).)  Let  { Zt ;  t  =  0,  ±1,±2,  •••}  be  a 
real-valued  zero-mean  stationary  Gaussian  process  with  normalized  spectral 
measure  ct2(-).  U  a~  has  at  least  two  atoms  in  [0,7r],  then  <r^({0})  >  0.  That 
is,  the  spectrum  of  {dt}  has  a  jump  at  0. 

Thus,  two  or  more  atoms  (jumps)  in  the  spectrum  of  {Ztj  prevent  the  ZCR. 
from  converging  to  a  constant  a.s.,  though  it  always  converges  a.s.  to  a. 
random  variable  because  of  the  assumed  Gaussianity  and  hence  strict  sta- 
tionarity.  Hence,  by  Theorem  3.2  the  spectrum  of  {(Xt  —  .5)(Xt_i  —  .5)}  does 
not  contain  a  clustered  difference  in  the  critical  intervals  (—2~n,2~n)  \  {0}), 
(— 2w,  —27 r  +  2_n),  (2%  —  2~n,2ie).  On  the  other  hand,  in  the  representation 
of  {Ah}  the  (j  do  not  have  constant  amplitudes  as  in  (  10). 

In  the  case  of  a  single  pure  atom,  the  ZCR  converges  almost  surely  to  a 
constant,  so  that  by  Theorem  3.2,  (  10)  holds. 

4  Mixed  Spectrum  Case 

The  case  of  practical  interest  is  a  finite  sum  of  complex  exponentials  plus 
continuous  spectrum  noise.  The  argument  proceeds  in  exactly  the  same  way 
as  before.  Suppose,  for  uij  E  (— 7r,  7 r],  ,  j  —  0, 1,  •  •  •  ,  p, 

Xt-m  =  Yje^t^  +  tt 
3=0 

is  weakly  stationary.  Further,  suppose  the  noise  et  is  zero  mean  ergodic  inde- 
pendent  of  the  zero  mean  (j.  When  forming  the  product  (Xt+k—m)(Xt  —  m), 
the  sample  covariance  of  the  cross  terms  (and  their  conjugates)  converges  to 
zero  with  probability  one  (by  zero  mean  ergodicity).  Since  the  noise  e  is 
ergodic,  its  sample  autocovariance  also  converges  almost  surely  to  its  auto¬ 
covariance.  Hence,  a  simple  consequence  of  the  results  of  the  previous  section 
as  well  as  of  independence,  we  have. 
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Theorem  4.1  Suppose,  for  u)j  £  (— 7r, 7r],  j  =  0, 1  ,■■■  ,p, 


Xt  —  m  —  ^  etw,t£j  T  e< 
j=o 


is  weakly  stationary.  Let  the  £ j  —  ^ j  =  0, 1,  •  ■  •  ,p,  be  zero  mean 
and  independent  of  the  zero  mean  ergodic  process  t.  Then, 

1  £(Xt+k  -  m)(Xt-m)  “-4-  R(k),  k  -  0,  ±1,  •  •  • 

iV  t= i 


Example  2.  Let  {4Y*}  be  as  in  Example  1,  and  consider  the  model  of  signal 
plus  noise 

Lj  =  A  t  +  )  t  =  0,  ±1,  •  •  • 

where  {ej  is  a  linear  process, 

oo  oo 

{«J  ~IID(0,<t2),  X]  Kl  <  00 

j—— OO  i=-oo 

independent  of  {JYt}.  Then  {e{}  is  a  continuous  spectrum  strictly  stationary 
ergodic  process  (Hannan  (1970),  p.  204).  By  Theorem  4.1  then 

l  SL 

R(k),  for  all  k 

^  t= i 

Filtering  { F< }  by  a  time  invariant  linear  filter  gives  again  a  discrete  spec¬ 
trum  signal  plus  an  independent  ergodic  stationary  noise  component,  so  that 
again  the  sample  autocovariance  converges  a.s.  to  the  true  autocovariance  for 
all  lags  k.  In  particular,  let  Ca  be  such  a  (parametric)  filter  with  an  impulse 
response  hj(a)  depending  on  a  parameter  a  £  A,  where  A  is  an  open  set. 
Define, 

OO 

Yt(a)=C„Yt  =  Y,h,(«)Yt-j 

3=0 

If  there  are  constants  Cj  >  0  such  that  J2jcj  <  oo  and  \hj(o)\  <  Cj  for 
all  j  and  uniformly  in  a ,  then  the  almost  sure  convergence  of  the  sample 
autocovariance  is  also  uniform  in  a  £  A  (Li  and  Kedem  (1993)). 
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